2-d Mesh-based Mosaic Representation for Manipulation of Video Objects with Occlusion

We present a 2-D mesh-based mosaic representation, consisting of an object mesh and a mosaic mesh for each frame and a nal mosaic image, for video objects with mildly deformable motion in the presence of self and/or object-to-object (external) occlusion. Unlike classical mosaic representations where successive frames are registered using global motion models, we map the uncovered regions in the successive frames onto the mosaic reference frame using local a ne models, i.e., those of the neighboring mesh patches. The proposed method to compute this mosaic representation is tightly coupled with an occlusion adaptive 2-D mesh tracking procedure, which consist of propagating the object mesh frame to frame, and updating of both object and mosaic meshes to optimize texture mapping from the mosaic to each instance of the object. The proposed representation has been applied to video object rendering and editing, including self trans guration, synthetic trans guration, and 2-D augmented reality in the presence of self and/or external occlusion. We also provide an algorithm to determine the minimum number of still views needed to reconstruct a replacement mosaic which is needed for synthetic trans guration. Experimental results are provided to demonstrate both the 2-D mesh-based mosaic synthesis and two di erent video object editing applications on real video sequences.